Evaluating Multi-Agent Robotic Systems Using Ground Truth
نویسندگان
چکیده
A thorough empirical evaluation of multi-agent robotic systems is greatly facilitated if the true state of the world over time can be obtained. The accuracy of the beliefs as well as the overall performance can then be measured objectively and efficiently. In this paper we present a system for determining the ground truth state of the world, similar to the ceiling cameras used in RoboCup small-size league. We have used this ground truth data to evaluate the accuracy of the selfand object-localization of the robots in our RoboCup mid-size league team, the AGILO RoboCuppers. More complex models of the state estimation module have also been learned. These models provide insight into the workings and shortcomings of this module, and can be used to improve it.
منابع مشابه
A data set for evaluating the performance of multi-class multi-object video tracking
One of the challenges in evaluating multi-object video detection, tracking and classification systems is having publically available data sets with which to compare different systems. However, the measures of performance for tracking and classification are different. Data sets that are suitable for evaluating tracking systems may not be appropriate for classification. Tracking video data sets t...
متن کاملLearning GP-BayesFilters via Gaussian process latent variable models
GP-BayesFilters are a general framework for integrating Gaussian process prediction and observation models into Bayesian filtering techniques, including particle filters and extended and unscented Kalman filters. GPBayesFilters have been shown to be extremely well suited for systems for which accurate parametric models are difficult to obtain. GP-BayesFilters learn non-parametric models from tr...
متن کاملExperiment Design for Large Multi-Robot Systems
Multi-robot systems present unique challenges for the experimenter. There are three key components for conducting successful multi-robot experiments: 1) Defining platformindependent metrics for evaluating algorithm performance, 2) determining the ground truth of the robot’s physical configuration and network connectivity, and 3) designing a usable system, with centralized data collection and ha...
متن کاملIncentivizing Evaluation via Limited Access to Ground Truth: Peer-Prediction Makes Things Worse
In many settings, an effective way of evaluating objects of interest is to collect evaluations from dispersed individuals and to aggregate these evaluations together. Some examples are categorizing online content and evaluating student assignments via peer grading. For this data science problem, one challenge is to motivate participants to conduct such evaluations carefully and to report them h...
متن کاملBenchmarks for Robotic Soccer Vision
Robotic soccer vision has been a major research problem in RoboCup and, even though many progresses have been made so that, for example, games now can run without many constraints on the lighting conditions, the problem has not been completely solved and on-site camera calibration is always a major activity for RoboCup soccer teams. While different robotic soccer vision and object perception te...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2004